QANCODE

Automated teting tool that required manual review

History

Problem

Coding all of the tasks

Solution

Take screenshot and output difference

Image diff internals

• matrix of numbers

• pad smaller image to make same size as larger image

• subtract second image from first image

• sum result

• if value exceeds threshold convert diff matrix back to image

import numpy as np
import cv2

h1 = cv2.imread('jupyter_notebooks/keenan/home1.png')
h2 = cv2.imread('jupyter_notebooks/keenan/home2.png')

Home 1

print(h1.ravel()[:500])

[59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59
 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37
 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16
 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59
 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37
 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16
 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59
 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37
 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16
 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59
 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37
 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16
 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59
 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37
 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16
 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59
 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37
 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16
 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59
 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37]

Home 2

print(h2.ravel()[:500])

[59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59
 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37
 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16
 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59
 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37
 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16
 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59
 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37
 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16
 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59
 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37
 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16
 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59
 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37
 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16
 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59
 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37
 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16
 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59
 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37 16 59 37]

diff = cv2.subtract(h1, h2)
diff = cv2.addWeighted(h1, 0.3, diff, 1, 0)
cv2.imwrite('jupyter_notebooks/keenan/diff.png', diff)

True

• Filters noise that requires manual review
• Up to QAer to decide whether change is expected or not

Example

Load QANCODE

import qancode

Instantiate QANCODE object with release candidate URL

qa = qancode.QANCODE(rc_url='https://test.encodedcc.org')

QANCODE methods

qa.list_methods()

check_downloads
check_permissions
check_requests
check_response_time
check_tools
check_trackhubs
compare_facets
find_differences
list_methods
show_differences

1. Crawl website and scrape/download data

• compare_facets
• check_downloads

2. Compare screenshots between production and RC

• find_differences
• check_permissions
• check_trackhubs

3. Submit RC to external (non-browser) requests

• check_requests
• check_tools
• check_response_time

Compare facets

Specify

• List of browsers
• List of users
• List of facets to check (item_types)
• Browser comparison (True/False)

Outputs

• Blue - match
• Red - same facet term, different value
• Yellow - facet term found on one list but not other

qa.compare_facets(
    browsers=['Safari'],
    users=['Public'],
    item_types=['/search/?type=Experiment'],
    browser_comparison=False
)

Opening https://www.encodeproject.org in Safari
Getting type: /search/?type=Experiment
Search page detected
Opening https://test.encodedcc.org in Safari
Getting type: /search/?type=Experiment
Search page detected

---------------------------------- Experiment ----------------------------------
Comparing data between URLs.
As user: Public
Browser: Safari
First URL: https://www.encodeproject.org
Second URL: https://test.encodedcc.org
Item type: /search/?type=Experiment
Assay
     MATCH
Assay category
     MATCH
Audit category
     MATCH
Audit category2
     MATCH
Audit category3
     MATCH
Available data
     MATCH
Biosample treatment
     MATCH
Biosample type
     MATCH
Date Submitted
     MATCH
Date released
     MATCH
Experiment status
     MATCH
Genome assembly (visualization)
     MATCH
Lab
     MATCH
Library depleted in
     MATCH
Library insert size (nt)
     MATCH
Library made from
     MATCH
Library treatment
     MATCH
Life stage
     MATCH
Mapped read length (nt)
     MATCH
Organ
     MATCH
Organism
     MATCH
Platform
     MATCH
Project
     MATCH
RFA
     MATCH
Read length (nt)
     MATCH
Replication type
     MATCH
Run type
     MATCH
Target of assay
     MATCH

qa.compare_facets_default_actions

['/search/?type=Experiment',
 '/search/?type=File',
 '/search/?type=Library',
 '/search/?type=AntibodyLot',
 '/search/?type=Biosample',
 '/search/?type=Donor',
 'search/?type=GeneticModification',
 '/search/?type=FileSet',
 '/search/?type=Annotation',
 '/search/?type=Series',
 '/search/?type=OrganismDevelopmentSeries',
 '/search/?type=UcscBrowserComposite',
 '/search/?type=ReferenceEpigenome',
 '/search/?type=Project',
 '/search/?type=ReplicationTimingSeries',
 '/search/?type=PublicationData',
 '/search/?type=MatchedSet',
 '/search/?type=TreatmentConcentrationSeries',
 '/search/?type=TreatmentTimeSeries',
 '/search/?type=Target',
 '/search/?type=Pipeline',
 '/search/?type=Publication',
 '/search/?type=Software',
 '/matrix/?type=Experiment',
 '/matrix/?type=Annotation']

Find differences

Specify

• List of browsers
• List of users
• List of pages (item_types) and actions (click_paths) to perform before taking screenshot

Outputs

• Pixel difference and image (image_diff folder on Desktop)

Tips

• item_types='random' checks three random items from the default action list
• qa.show_differences() to display image difference in Jupyter notebook

qa.find_differences(
    browsers=['Safari'],
    users=['Public'],
    item_types=['/']
)

Opening https://www.encodeproject.org in Safari
Loading complete
Taking picture of https://www.encodeproject.org/
/var/folders/v8/vsjvg3vn5mz0nrhvkpp7f0_40000gp/T/tmpwtnsuf3t/prod218239331008719121818735299822922991447.png
Opening https://test.encodedcc.org in Safari
Loading complete
Taking picture of https://test.encodedcc.org/
/var/folders/v8/vsjvg3vn5mz0nrhvkpp7f0_40000gp/T/tmpwtnsuf3t/RC16617693017015765867481778073000793216.png

Comparing screenshots between URLs.
As user: Public
Browser: Safari
First URL: https://www.encodeproject.org
Second URL: https://test.encodedcc.org
Item type: /
Click path: None
MATCH
Distance metric: 0

[(False, 'safari_FRONT_PAGE_public_None_prod_rc_diff.png')]

qa = qancode.QANCODE(rc_url='https://nyt.com')
qa.find_differences(
    browsers=['Safari'],
    users=['Public'],
    item_types=['/']
)

Opening https://www.encodeproject.org in Safari
Loading complete
Taking picture of https://www.encodeproject.org/
/var/folders/v8/vsjvg3vn5mz0nrhvkpp7f0_40000gp/T/tmp4cuvoiyo/prod175449858901618789148939899907510107446.png
Opening https://nyt.com in Safari
Loading complete
Taking picture of https://www.nytimes.com/
/var/folders/v8/vsjvg3vn5mz0nrhvkpp7f0_40000gp/T/tmp4cuvoiyo/RC122276463339994286958944422798482427672.png

Comparing screenshots between URLs.
As user: Public
Browser: Safari
First URL: https://www.encodeproject.org
Second URL: https://nyt.com
Item type: /
Click path: None
Difference detected
Outputting file safari_FRONT_PAGE_public_None_prod_rc_diff.png
Distance metric: 302986600

[(True, 'safari_FRONT_PAGE_public_None_prod_rc_diff.png')]

qa.show_differences() 

safari_FRONT_PAGE_public_None_prod_rc_diff.png:

Extensions

• Specify task to perform before taking picture

qa.find_differences_default_actions[:5]

[('/', None),
 ('/targets/?status=deleted', None),
 ('/antibodies/?status=deleted', None),
 ('/search/?type=Biosample&status=deleted', None),
 ('/experiments/ENCSR000CWD/', None)]

qa.check_trackhubs_default_actions[:5]

[('/experiments/ENCSR502NRF/', qancode.clickpaths.OpenUCSCGenomeBrowserGRCh38),
 ('/experiments/ENCSR502NRF/', qancode.clickpaths.OpenUCSCGenomeBrowserHG19),
 ('/experiments/ENCSR985KAT/', qancode.clickpaths.OpenUCSCGenomeBrowserHG19),
 ('/experiments/ENCSR426UUG/', qancode.clickpaths.OpenUCSCGenomeBrowserGRCh38),
 ('/experiments/ENCSR293WTN/', qancode.clickpaths.OpenUCSCGenomeBrowserMM9)]

Repo

def perform_action(self):
        for button in self.driver.find_elements_by_tag_name(ExperimentPage.all_buttons_tag_name):
            try:
                if button.text == 'Visualize':
                    button.click()
                    self.driver.wait.until(
                        EC.element_to_be_clickable(
                            (By.CLASS_NAME, VisualizeModal.modal_class)
                        )
                    )
                    break
            except:
                pass
        modal = self.driver.wait.until(
            EC.element_to_be_clickable(
                (By.CLASS_NAME, VisualizeModal.modal_class)
            )
        )
        UCSC_links = modal.find_elements_by_partial_link_text(
            VisualizeModal.UCSC_link_partial_link_text
        )
        for link in UCSC_links:
            if self.assembly in link.get_attribute('href'):
                print('Opening genome browser')
                link.click()
                break
        self.driver.wait.until(
            EC.element_to_be_clickable(
                (By.ID, UCSCGenomeBrowser.zoom_one_id)
            )
        )

from qancode.clickpaths import OpenUCSCGenomeBrowserHG19

qa.find_differences(
    browsers=['Safari'],
    users=['Public'],
    item_types=['/experiments/ENCSR324UCN/'],
    click_paths=[OpenUCSCGenomeBrowserHG19]
)

Easy problems with QANCODE

• metadata updates

Hard problems with QANCODE

• browser updates
• portal DOM updates